Many computing devices, such as smartphones, desktops, laptops, tablets, game consoles, and the like, utilize automatic speech recognition (ASR) for performing a number of tasks including voice search and short message dictation. In an effort to improve the accuracy of ASR, the use of deep neural networks (DNNs) has been proposed. DNNs are artificial neural networks with more than one hidden layer between input and output layers and may model complex non-linear relationships. The hidden layers in DNNs provide additional levels of abstraction, thus increasing its modeling capability. DNNs when utilized in ASR however, suffer from a number of drawbacks associated with adaption and personalization. For example, the use of DNNs, while increasing ASR accuracy, also is accompanied by a very large number of parameters making the adaptation of DNN models very challenging. Furthermore, the cost associated with using DNNs in personalized ASR applications (i.e., multiple individual speakers) is prohibitive due to the need to store very large DNN models for each individual speaker during deployment. It is with respect to these considerations and others that the various embodiments of the present invention have been made.